Summary

Introduce CompactionTaskTracker to provide full lifecycle observability for base/cumulative/full compaction tasks. This PR adds two query channels:

1. information_schema.be_compaction_tasks system table (38 columns) — query via SQL across all BEs
2. GET /api/compaction/profile HTTP API — query via JSON on a single BE, with filtering support

What problem does this PR solve?

Currently, compaction execution metrics (input/output data sizes, row counts, merge latency, etc.) are lost once the compaction object is destructed. Operators have no way to inspect historical compaction performance — only the current running status is available. Users also cannot see pending or running compaction task details through SQL.

This PR solves both problems with a unified CompactionTaskTracker mechanism that tracks compaction tasks across their full lifecycle: PENDING → RUNNING → FINISHED/FAILED.

Architecture

                         ┌── Push ──┐
  Compaction entry points ──→ CompactionTaskTracker (in-memory) ←── Push ── Compaction base class
   (register_task)            │  _active_tasks (PENDING/RUNNING)          (complete/fail)
                              │  _completed_tasks (FINISHED/FAILED)
                              │
               ┌──── Pull ────┴──── Pull ────┐
               ↓                              ↓
   SchemaCompactionTasksScanner     CompactionProfileAction
    get_all_tasks() snapshot         get_completed_tasks() filtered
               ↓                              ↓
   FE fan-out to all BEs → SQL      JSON HTTP Response

• Push-based collection: Compaction entry points and execution layer push task info at lifecycle boundaries
• Pull-based querying: System table scanner and HTTP action pull snapshots with read locks, never blocking compaction
• Pure in-memory storage: Two containers, no persistence. Data clears on BE restart.

System Table: information_schema.be_compaction_tasks

Schema (38 columns)

SQL Query Examples

-- View all running compaction tasks across all BEs
SELECT BACKEND_ID, TABLET_ID, COMPACTION_TYPE, ELAPSED_TIME_MS, INPUT_DATA_SIZE
FROM information_schema.be_compaction_tasks
WHERE STATUS = 'RUNNING'
ORDER BY ELAPSED_TIME_MS DESC;

-- Track vertical compaction progress for large compactions
SELECT BACKEND_ID, TABLET_ID, COMPACTION_TYPE,
       VERTICAL_COMPLETED_GROUPS, VERTICAL_TOTAL_GROUPS,
       CONCAT(ROUND(VERTICAL_COMPLETED_GROUPS * 100.0 / NULLIF(VERTICAL_TOTAL_GROUPS, 0), 1), '%') AS progress,
       ELAPSED_TIME_MS
FROM information_schema.be_compaction_tasks
WHERE STATUS = 'RUNNING' AND IS_VERTICAL = true;

-- Find the slowest finished compactions
SELECT TABLET_ID, COMPACTION_TYPE, ELAPSED_TIME_MS, INPUT_DATA_SIZE, OUTPUT_DATA_SIZE,
       PEAK_MEMORY_BYTES, BYTES_READ_FROM_REMOTE
FROM information_schema.be_compaction_tasks
WHERE STATUS = 'FINISHED'
ORDER BY ELAPSED_TIME_MS DESC LIMIT 10;

-- Compaction statistics by trigger method
SELECT TRIGGER_METHOD, COUNT(*) AS cnt
FROM information_schema.be_compaction_tasks
WHERE STATUS = 'FINISHED' GROUP BY TRIGGER_METHOD;

-- Check remote IO ratio (disaggregated storage troubleshooting)
SELECT TABLET_ID, COMPACTION_TYPE, BYTES_READ_FROM_LOCAL, BYTES_READ_FROM_REMOTE,
       ROUND(BYTES_READ_FROM_REMOTE * 100.0 / NULLIF(BYTES_READ_FROM_LOCAL + BYTES_READ_FROM_REMOTE, 0), 2) AS remote_pct
FROM information_schema.be_compaction_tasks
WHERE STATUS = 'FINISHED' AND BYTES_READ_FROM_REMOTE > 0
ORDER BY remote_pct DESC;

-- View compaction history for a specific table
SELECT * FROM information_schema.be_compaction_tasks
WHERE TABLE_ID = 12345 ORDER BY SCHEDULED_TIME DESC LIMIT 20;

HTTP API: GET /api/compaction/profile

Queries completed compaction profiles on a single BE with optional filtering. All parameters can be combined (AND logic).

Request Parameters

Response Example

{
  "status": "Success",
  "compaction_profiles": [
    {
      "compaction_id": 487,
      "compaction_type": "cumulative",
      "tablet_id": 12345,
      "table_id": 67890,
      "partition_id": 11111,
      "trigger_method": "AUTO",
      "compaction_score": 10,
      "scheduled_time": "2025-07-15 14:02:30",
      "start_time": "2025-07-15 14:02:31",
      "end_time": "2025-07-15 14:02:31",
      "cost_time_ms": 236,
      "success": true,
      "input_rowsets_count": 5,
      "input_row_num": 52000,
      "input_data_size": 10706329,
      "input_index_size": 204800,
      "input_total_size": 10911129,
      "input_segments_num": 5,
      "input_version_range": "[12-16]",
      "merged_rows": 1200,
      "filtered_rows": 50,
      "output_rows": 50750,
      "output_row_num": 50750,
      "output_data_size": 5033164,
      "output_index_size": 102400,
      "output_total_size": 5135564,
      "output_segments_num": 1,
      "output_version": "[12-16]",
      "merge_latency_ms": 180,
      "bytes_read_from_local": 10706329,
      "bytes_read_from_remote": 0,
      "peak_memory_bytes": 33554432,
      "is_vertical": true,
      "permits": 10706329,
      "vertical_total_groups": 4,
      "vertical_completed_groups": 4
    }
  ]
}

Failed compactions additionally include "status_msg": "error message".

curl Examples

# Recent 10 compaction profiles
curl "http://BE_HOST:BE_HTTP_PORT/api/compaction/profile?top_n=10"

# A specific tablet's recent 5 compactions
curl "http://BE_HOST:BE_HTTP_PORT/api/compaction/profile?tablet_id=12345&top_n=5"

# Only failed base compactions
curl "http://BE_HOST:BE_HTTP_PORT/api/compaction/profile?compact_type=base&success=false"

# Combined: tablet's recent 3 failed cumulative compactions
curl "http://BE_HOST:BE_HTTP_PORT/api/compaction/profile?tablet_id=12345&top_n=3&compact_type=cumulative&success=false"

Configuration

Scope

• Tracked: base / cumulative / full compaction (local + cloud mode)
• Not tracked: single_replica, cold_data, index_change compaction (low frequency, limited diagnostic value)
• Trigger methods: AUTO (background scheduling), MANUAL (HTTP API), LOAD_TRIGGERED (load-time cumulative)

Test plan

• 15 unit tests (CompactionTaskTrackerTest.*) covering full lifecycle, failure paths, TRY_LOCK_FAILED cleanup, vertical progress, concurrent safety, config switch, filters
• Regression test: test_be_compaction_tasks (system table)
• Regression test: test_compaction_profile_action (HTTP API)